A Comparative Evaluation of QA Systems over List Questions
نویسندگان
چکیده
The evaluation of a Question Answering system is a challenging task. In this paper we evaluate our system, LX-ListQuestion, a Web-based QA System that focuses on answering list questions. We compare our system against other QA Systems and the results were analyzed in two ways: (i) the quantitative evaluation of answers provides recall, precision and F-measure and (ii) the question coverage that indicate the usefulness of the system to the user by counting the number of questions for which the system provides at least one correct answer. The evaluation brings interesting results that points to a certain degree of complementary between different approaches.
منابع مشابه
A New Statistical Model for Evaluation Interactive Question Answering Systems Using Regression
The development of computer systems and extensive use of information technology in the everyday life of people have just made it more and more important for them to make quick access to information that has received great importance. Increasing the volume of information makes it difficult to manage or control. Thus, some instruments need to be provided to use this information. The QA system is ...
متن کاملOverview of the TREC 2003 Question Answering Track
The TREC 2003 question answering track contained two tasks, the passages task and the main task. In the passages task, systems returned a single text snippet in response to factoid questions; the evaluation metric was the number of snippets that contained a correct answer. The main task contained three separate types of questions, factoid questions, list questions, and definition questions. Eac...
متن کاملA Comparative Analysis: QA Evaluation Questions versus Real-world Queries
This paper presents a comparative analysis of user queries to a web search engine, questions to a Q&A service (answers.com), and questions employed in question answering (QA) evaluations at TREC and CLEF. The analysis shows that user queries to search engines contain mostly content words (i.e. keywords) but lack structure words (i.e. stopwords) and capitalization. Thus, they resemble natural la...
متن کاملFDUQA on TREC 2005 QA Track
In this year’s QA Track, we participant in the main and document ranking task and do not take part in the relation task. We put the most effort in factoid and definition questions, and very little on list questions and document ranking task. For factoid questions, we use three QA systems: system 1, system 2 and system 3. System 1 is very similar to our last year’s system [Wu et al, 2004] except...
متن کاملNeural Domain Adaptation for Biomedical Question Answering
Factoid question answering (QA) has recently benefited from the development of deep learning (DL) systems. Neural network models outperform traditional approaches in domains where large datasets exist, such as SQuAD (≈ 100, 000 questions) for Wikipedia articles. However, these systems have not yet been applied to QA in more specific domains, such as biomedicine, because datasets are generally t...
متن کامل